Extensions | 您所在的位置:网站首页 › scrapy pipeline from_crawler › Extensions |
Extensions露
The extensions framework provides a mechanism for inserting your own custom functionality into Scrapy. Extensions are just regular classes. Extension settings露Extensions use the Scrapy settings to manage their settings, just like any other Scrapy code. It is customary for extensions to prefix their settings with their own name, to avoid collision with existing (and future) extensions. For example, a hypothetical extension to handle Google Sitemaps would use settings like GOOGLESITEMAP_ENABLED, GOOGLESITEMAP_DEPTH, and so on. Loading & activating extensions露Extensions are loaded and activated at startup by instantiating a single instance of the extension class per spider being run. All the extension initialization code must be performed in the class __init__ method. To make an extension available, add it to the EXTENSIONS setting in your Scrapy settings. In EXTENSIONS, each extension is represented by a string: the full Python path to the extension鈥檚 class name. For example: EXTENSIONS = { "scrapy.extensions.corestats.CoreStats": 500, "scrapy.extensions.telnet.TelnetConsole": 500, }As you can see, the EXTENSIONS setting is a dict where the keys are the extension paths, and their values are the orders, which define the extension loading order. The EXTENSIONS setting is merged with the EXTENSIONS_BASE setting defined in Scrapy (and not meant to be overridden) and then sorted by order to get the final sorted list of enabled extensions. As extensions typically do not depend on each other, their loading order is irrelevant in most cases. This is why the EXTENSIONS_BASE setting defines all extensions with the same order (0). However, this feature can be exploited if you need to add an extension which depends on other extensions already loaded. Available, enabled and disabled extensions露Not all available extensions will be enabled. Some of them usually depend on a particular setting. For example, the HTTP Cache extension is available by default but disabled unless the HTTPCACHE_ENABLED setting is set. Disabling an extension露In order to disable an extension that comes enabled by default (i.e. those included in the EXTENSIONS_BASE setting) you must set its order to None. For example: EXTENSIONS = { "scrapy.extensions.corestats.CoreStats": None, } Writing your own extension露Each extension is a Python class. The main entry point for a Scrapy extension (this also includes middlewares and pipelines) is the from_crawler class method which receives a Crawler instance. Through the Crawler object you can access settings, signals, stats, and also control the crawling behaviour. Typically, extensions connect to signals and perform tasks triggered by them. Finally, if the from_crawler method raises the NotConfigured exception, the extension will be disabled. Otherwise, the extension will be enabled. Sample extension露Here we will implement a simple extension to illustrate the concepts described in the previous section. This extension will log a message every time: a spider is opened a spider is closed a specific number of items are scraped The extension will be enabled through the MYEXT_ENABLED setting and the number of items will be specified through the MYEXT_ITEMCOUNT setting. Here is the code of such extension: import logging from scrapy import signals from scrapy.exceptions import NotConfigured logger = logging.getLogger(__name__) class SpiderOpenCloseLogging: def __init__(self, item_count): self.item_count = item_count self.items_scraped = 0 @classmethod def from_crawler(cls, crawler): # first check if the extension should be enabled and raise # NotConfigured otherwise if not crawler.settings.getbool("MYEXT_ENABLED"): raise NotConfigured # get the number of items from settings item_count = crawler.settings.getint("MYEXT_ITEMCOUNT", 1000) # instantiate the extension object ext = cls(item_count) # connect the extension object to signals crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened) crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed) crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped) # return the extension object return ext def spider_opened(self, spider): logger.info("opened spider %s", spider.name) def spider_closed(self, spider): logger.info("closed spider %s", spider.name) def item_scraped(self, item, spider): self.items_scraped += 1 if self.items_scraped % self.item_count == 0: logger.info("scraped %d items", self.items_scraped) Built-in extensions reference露 General purpose extensions露 Log Stats extension露 class scrapy.extensions.logstats.LogStats[source]露Log basic stats like crawled pages and scraped items. Core Stats extension露 class scrapy.extensions.corestats.CoreStats[source]露Enable the collection of core statistics, provided the stats collection is enabled (see Stats Collection). Telnet console extension露 class scrapy.extensions.telnet.TelnetConsole[source]露Provides a telnet console for getting into a Python interpreter inside the currently running Scrapy process, which can be very useful for debugging. The telnet console must be enabled by the TELNETCONSOLE_ENABLED setting, and the server will listen in the port specified in TELNETCONSOLE_PORT. Memory usage extension露 class scrapy.extensions.memusage.MemoryUsage[source]露Note This extension does not work in Windows. Monitors the memory used by the Scrapy process that runs the spider and: sends a notification e-mail when it exceeds a certain value closes the spider when it exceeds a certain value The notification e-mails can be triggered when a certain warning value is reached (MEMUSAGE_WARNING_MB) and when the maximum value is reached (MEMUSAGE_LIMIT_MB) which will also cause the spider to be closed and the Scrapy process to be terminated. This extension is enabled by the MEMUSAGE_ENABLED setting and can be configured with the following settings: MEMUSAGE_LIMIT_MB MEMUSAGE_WARNING_MB MEMUSAGE_NOTIFY_MAIL MEMUSAGE_CHECK_INTERVAL_SECONDS Memory debugger extension露 class scrapy.extensions.memdebug.MemoryDebugger[source]露An extension for debugging memory usage. It collects information about: objects uncollected by the Python garbage collector objects left alive that shouldn鈥檛. For more info, see Debugging memory leaks with trackref To enable this extension, turn on the MEMDEBUG_ENABLED setting. The info will be stored in the stats. Close spider extension露 class scrapy.extensions.closespider.CloseSpider[source]露Closes a spider automatically when some conditions are met, using a specific closing reason for each condition. The conditions for closing a spider can be configured through the following settings: CLOSESPIDER_TIMEOUT CLOSESPIDER_TIMEOUT_NO_ITEM CLOSESPIDER_ITEMCOUNT CLOSESPIDER_PAGECOUNT CLOSESPIDER_ERRORCOUNT Note When a certain closing condition is met, requests which are currently in the downloader queue (up to CONCURRENT_REQUESTS requests) are still processed. CLOSESPIDER_TIMEOUT露Default: 0 An integer which specifies a number of seconds. If the spider remains open for more than that number of second, it will be automatically closed with the reason closespider_timeout. If zero (or non set), spiders won鈥檛 be closed by timeout. CLOSESPIDER_TIMEOUT_NO_ITEM露Default: 0 An integer which specifies a number of seconds. If the spider has not produced any items in the last number of seconds, it will be closed with the reason closespider_timeout_no_item. If zero (or non set), spiders won鈥檛 be closed regardless if it hasn鈥檛 produced any items. CLOSESPIDER_ITEMCOUNT露Default: 0 An integer which specifies a number of items. If the spider scrapes more than that amount and those items are passed by the item pipeline, the spider will be closed with the reason closespider_itemcount. If zero (or non set), spiders won鈥檛 be closed by number of passed items. CLOSESPIDER_PAGECOUNT露Default: 0 An integer which specifies the maximum number of responses to crawl. If the spider crawls more than that, the spider will be closed with the reason closespider_pagecount. If zero (or non set), spiders won鈥檛 be closed by number of crawled responses. CLOSESPIDER_ERRORCOUNT露Default: 0 An integer which specifies the maximum number of errors to receive before closing the spider. If the spider generates more than that number of errors, it will be closed with the reason closespider_errorcount. If zero (or non set), spiders won鈥檛 be closed by number of errors. StatsMailer extension露 class scrapy.extensions.statsmailer.StatsMailer[source]露This simple extension can be used to send a notification e-mail every time a domain has finished scraping, including the Scrapy stats collected. The email will be sent to all recipients specified in the STATSMAILER_RCPTS setting. Emails can be sent using the MailSender class. To see a full list of parameters, including examples on how to instantiate MailSender and use mail settings, see Sending e-mail. Periodic log extension露 class scrapy.extensions.periodic_log.PeriodicLog[source]露This extension periodically logs rich stat data as a JSON object: 2023-08-04 02:30:57 [scrapy.extensions.logstats] INFO: Crawled 976 pages (at 162 pages/min), scraped 925 items (at 161 items/min) 2023-08-04 02:30:57 [scrapy.extensions.periodic_log] INFO: { "delta": { "downloader/request_bytes": 55582, "downloader/request_count": 162, "downloader/request_method_count/GET": 162, "downloader/response_bytes": 618133, "downloader/response_count": 162, "downloader/response_status_count/200": 162, "item_scraped_count": 161 }, "stats": { "downloader/request_bytes": 338243, "downloader/request_count": 992, "downloader/request_method_count/GET": 992, "downloader/response_bytes": 3836736, "downloader/response_count": 976, "downloader/response_status_count/200": 976, "item_scraped_count": 925, "log_count/INFO": 21, "log_count/WARNING": 1, "scheduler/dequeued": 992, "scheduler/dequeued/memory": 992, "scheduler/enqueued": 1050, "scheduler/enqueued/memory": 1050 }, "time": { "elapsed": 360.008903, "log_interval": 60.0, "log_interval_real": 60.006694, "start_time": "2023-08-03 23:24:57", "utcnow": "2023-08-03 23:30:57" } }This extension logs the following configurable sections: "delta" shows how some numeric stats have changed since the last stats log message. The PERIODIC_LOG_DELTA setting determines the target stats. They must have int or float values. "stats" shows the current value of some stats. The PERIODIC_LOG_STATS setting determines the target stats. "time" shows detailed timing data. The PERIODIC_LOG_TIMING_ENABLED setting determines whether or not to show this section. This extension logs data at the start, then on a fixed time interval configurable through the LOGSTATS_INTERVAL setting, and finally right before the crawl ends. Example extension configuration: custom_settings = { "LOG_LEVEL": "INFO", "PERIODIC_LOG_STATS": { "include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count/"], }, "PERIODIC_LOG_DELTA": {"include": ["downloader/"]}, "PERIODIC_LOG_TIMING_ENABLED": True, "EXTENSIONS": { "scrapy.extensions.periodic_log.PeriodicLog": 0, }, } PERIODIC_LOG_DELTA露Default: None "PERIODIC_LOG_DELTA": True - show deltas for all int and float stat values. "PERIODIC_LOG_DELTA": {"include": ["downloader/", "scheduler/"]} - show deltas for stats with names containing any configured substring. "PERIODIC_LOG_DELTA": {"exclude": ["downloader/"]} - show deltas for all stats with names not containing any configured substring. PERIODIC_LOG_STATS露Default: None "PERIODIC_LOG_STATS": True - show the current value of all stats. "PERIODIC_LOG_STATS": {"include": ["downloader/", "scheduler/"]} - show current values for stats with names containing any configured substring. "PERIODIC_LOG_STATS": {"exclude": ["downloader/"]} - show current values for all stats with names not containing any configured substring. PERIODIC_LOG_TIMING_ENABLED露Default: False True enables logging of timing data (i.e. the "time" section). Debugging extensions露 Stack trace dump extension露 class scrapy.extensions.periodic_log.StackTraceDump露Dumps information about the running process when a SIGQUIT or SIGUSR2 signal is received. The information dumped is the following: engine status (using scrapy.utils.engine.get_engine_status()) live references (see Debugging memory leaks with trackref) stack trace of all threads After the stack trace and engine status is dumped, the Scrapy process continues running normally. This extension only works on POSIX-compliant platforms (i.e. not Windows), because the SIGQUIT and SIGUSR2 signals are not available on Windows. There are at least two ways to send Scrapy the SIGQUIT signal: By pressing Ctrl-while a Scrapy process is running (Linux only?) By running this command (assuming is the process id of the Scrapy process): kill -QUIT Debugger extension露 class scrapy.extensions.periodic_log.Debugger露Invokes a Python debugger inside a running Scrapy process when a SIGUSR2 signal is received. After the debugger is exited, the Scrapy process continues running normally. For more info see Debugging in Python. This extension only works on POSIX-compliant platforms (i.e. not Windows). |
CopyRight 2018-2019 实验室设备网 版权所有 |